Skip to content

Update default target-pod and inject it into response metadata #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Feb 3, 2025
Merged

Update default target-pod and inject it into response metadata #270

merged 4 commits into from
Feb 3, 2025

Conversation

ahg-g
Copy link
Contributor

@ahg-g ahg-g commented Jan 31, 2025

Fixes #258
Fixes #181

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 31, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 31, 2025
@netlify Netlify
Copy link

netlify bot commented Jan 31, 2025 •
edited
Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 91d1d4f
🔍 Latest deploy log https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67a11e4221bec50008738830
😎 Deploy Preview https://deploy-preview-270--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 31, 2025
liu-cong
liu-cong reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@liu-cong liu-cong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/hold
/lgtm

Just a few nits. Feel free to unhold.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 31, 2025
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 31, 2025
@liu-cong
Copy link
Contributor

Pls also update this

gateway-api-inference-extension/docs/proposals/003-model-server-protocol/protocol.md

Line 15 in f9d2ef9

adding the `target-pod` HTTP header in the request, or otherwise return an error.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 1, 2025
AndresGuedez
AndresGuedez reviewed Feb 1, 2025
View reviewed changes
pkg/ext-proc/handlers/request.go
@@ -118,6 +122,15 @@ func (s *Server) HandleRequestBody(reqCtx *RequestContext, req *extProcPb.Proces
},
},
},
DynamicMetadata: &structpb.Struct{
Fields: map[string]*structpb.Value{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM.

@yanavlasov PTAL as well.

@ahg-g
Copy link
Contributor Author

ahg-g commented Feb 3, 2025

Pls also update this

gateway-api-inference-extension/docs/proposals/003-model-server-protocol/protocol.md

Line 15 in f9d2ef9

adding the `target-pod` HTTP header in the request, or otherwise return an error.

done

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2025
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2025
@liu-cong
Copy link
Contributor

liu-cong commented Feb 3, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 3, 2025
robscott
robscott reviewed Feb 3, 2025
View reviewed changes
docs/proposals/003-endpoint-picker-protocol/README.md Outdated
@@ -12,7 +12,7 @@ The EPP MUST implement the Envoy
[external processing service](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ext_proc/v3/external_processor)protocol.

For each HTTP request, the EPP MUST communicate to the proxy the picked model server endpoint, via
adding the `target-pod` HTTP header in the request, or otherwise return an error.
adding the `x-gateway-destination-endpoint` HTTP header in the request and as an unstructured entry in the [dynamic_metadata](https://github.com/envoyproxy/go-control-plane/blob/c19bf63a811c90bf9e02f8e0dc1dcef94931ebb4/envoy/service/ext_proc/v3/external_processor.pb.go#L320) field of the ext-proc response, or otherwise return an error.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify - we're setting this endpoint information here in two places to ensure a wide variety of implementations can support this? What should an implementation do if both values are set but differ? Should one have precedent over the other?

Copy link
Contributor Author

@ahg-g ahg-g Feb 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify - we're setting this endpoint information here in two places to ensure a wide variety of implementations can support this?

Correct.

What should an implementation do if both values are set but differ? Should one have precedent over the other?

Good point, I think we need to say that they must be not differ, and if they do, the behavior is unknown, this is because some gateway providers will always only look at one of them, wdyt?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think that's reasonable, just wanted to make sure we had it written out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@k8s-ci-robot k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 3, 2025
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 3, 2025
@robscott
Copy link
Member

robscott commented Feb 3, 2025

Thanks @ahg-g!

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 3, 2025
@ahg-g
Copy link
Contributor Author

ahg-g commented Feb 3, 2025

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2025
@k8s-ci-robot k8s-ci-robot merged commit 614199e into kubernetes-sigs:main Feb 3, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@robscott robscott robscott left review comments

@liu-cong liu-cong liu-cong left review comments

@AndresGuedez AndresGuedez AndresGuedez left review comments

@kfswain kfswain Awaiting requested review from kfswain

Assignees

@robscott robscott

@liu-cong liu-cong

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
5 participants
@ahg-g @k8s-ci-robot @liu-cong @robscott @AndresGuedez